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ABSTRACT 

A group of 34 experimental and 45 control teachers 
provided data on 1^866 pupils in 75 different classrooms in 24 
communities in the United States to determine the extent to which the 
workshop training provided for recipients of the Sears-Roebuck 
Foundation Fellowships in the summer of 1972 affected the economic 
understanding of their pupils. Pupils in grades 2, 3, and 4 took the 
Primary Test of Economic Understanding while students in grades 5 and 
6 took the Test of Elementary Economics. All students took a fcrm of 
the Flanagan Test of General Ability. After teachers had completed 
their units in economics^ the economics tests were administered 
again. Change scores were computed for each pupil; mean pre-test^ 
post-testy change scores and mean scores were computed ,for each 
class. Each workshop group was tested separately. The results from 
the Des Moines group indicated that pupils studying under workshop 
teachers learned signif icantD.y more than similar pupils. In seven of 
the eight other workshop groups^ the pupils of the workshop teachers 
learned, more than similar pupils under teachers who had not attended 
workshops . (Author/KSM) 



THE IMPACT OF ECONOMICS WORKSHOPS 
FOR ELEMENTARY SCHOOL TEACHERS ON 
THE ECONOMIC UNDERSTANDING OF THEIR 
PUPILS 

George G. Dawson and Donald G. Davison 



The Joint: Council on Economic Education 

with the cooperation of 
The Iowa Council on Economic Education 
1973 



Table 10: Mean Gain (Unadjusted) By TreaCment: Croup, 
Grades 2-4 



Table 11 
Table 12 
Table 13 



Summary of c-cescs for Grades 2-4 
Summary of t-cescs for Grades 5-6 
Classroom Means 



SUMMARY OF REPORT 



This summary has been prepared for Chose interested only in the overall 
findings of the study and a brief discription of the research design. 

The purpose of this study was to determine the extent to which the 
workshop training provided for recipients of The Sears-Roebuck Foundation 
FelLowships in the summer of 1972 affected the economic understanding of 
their pupils. Fellowship recipients from nine workshops were selected for 
participation in the study. For each FelJowship recipient selected, one or 
more control teachers were chosen. Dich control teacher had to be teaching 
at the same grade Level, in the same general geographic area (but in a different 
school), and be of the same sex as the experimental teacher (Fellowship 
recipient). Efforts were made to match experimental and control teachers 
in terms of agi*, teaching experience, and economics backgrounds, except for 
the fact that the experimental teachers had attended 1972 summer workshops 
in economics. AJ though some of the teachers were unable to follow through 
as planned, a total of "i^i experimental and A5 control teachers did follow 
through and did prt)vide usable data on 1,866 pupils in 75 different classrooms 
in 2A communities in various parts of the United States. 

Karl.y in the fall semester of 1972, all pupils at the second, third, and 
fourth grade levels took the Prjiniar y Test of Economic Understanding (a stand- 
ardif'.ed test developed by the Towa Council on Economic F,ducatlon) and the 
appropriate form of the Flanagan Test of Genera l Ajpility. All pupils at the 
fifth and sixth grade levels took the '^'f;,^,^:^ Q, [_ AL^-"V^-Jlf-*jLrX „ A^9.l\*^"^ (developed 
by the West Springfield, Massachusetts, Developmental Economic Education 
Project) and tlie appropriate form of the Flanagan ^pj^j^^oj^ J^^J\9J,'j-L^^ll^ JAky * 
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Tlio tests were administered by paid proctors, so that the experimental and 
control tencliers would not be inclined to "teach to the lost" and prejudice 
tlie results. Test papers were sent to the Joint Council on Kconomic lulucation 
and to tiie Iowa Council on Economic Education for scoring and analysis. 

After the experimental and control teachers iiad completed their units 
in economics (or toward the end of the school year in the case of those 
rcportedJy integrating economics into the curriculum throughout), the economics 
tests were again administered to the pupils. Change scores (the difference 
between the pre-test score and the post-test score) were computed for each 
pupil. Mean pre-test scores, mean post-test scores, mean change scores, and 
mean scores on tiie Test of Gen er al Ability were computed for eacii class. Mean 
ciiange scores -were adjusted to account for differences in pupil ability as 
measured by the Te st of > G e n o r a i Al) i 1 i t y , In the case of teadiors from the 
Des Moines workshop (17 of the experimental teachers), the possible impact of 
different textbook materials being used by the pupils was also taken into account. 

Because each workshop was different in terms of content and approach, each 
was analyzed separately. The Des Moines group accounted for about half of the 
teacher and pupil population sample involved In the study and presented the 
best situation from the point of view of control. The results in Des Moines 
indicated that pupils studying under workshop teachers learned significantly 
more than similar pupils, using the same materials, and being taught by teachers 
with similar backgrounds but without the workshop experience, in seven of the 
eight other workshop groups, the pupils of the workshop teachers learned more 
than similar pupils studying under teachers who had not attended workshops. 

Although the (experimental teachers wore not selected at random, they 
app<»ar to have Ixu'n repri^seni at i v(» of the entire group of Kellowshlp recipients, 

i L 



at lease as measured by per fovnuince on Che SRA Test of Econ omic Under- 
scandinj'.. The SRA cest was adminlscered Co ail Fellowship recipients boch 
before and a£cer t he workshops. The mean posc-cesc score and standard deviacion 
for the expeiimenial teachers was praccically idencical with the mean post- 
test score and standard deviation oE the Fellowship recipients as a whole. 

ft Is poscJibio, of course, that the superior results obtained by the 
Fellowship recipients are accounted for by motivation or some other factor 
not considered in this study. Recognizing the difficulty of measuring moti- 
vation and other possibly significant variables, the researchers tentatively 
conclude that the workshop training provided during the summer of 1972 for 
recipients of The Sears-Roebuck Foundation Fellowships did have a positive 
impact on the pupils of those te.ichers and did result in greater learning 
as compared with similar pupils, in similar situacionsj taught by similar 
te.achers without workshop training. 
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Introduccion 

In the spring of 1972 the Joint Council on Economic Education submitted 
to The Sears--Roebuck Foundation a proposal for evaluating the effectiveness 
ut the Fellowship Program for Elementary School Teachers. A two-phase 
evaluation scheme was developed. Phase One was designed to answer the 
question: "What was the impact of the workshop training on the teachers?" 
This included the administration of a standardized test of economics (the 
lesL_ of Economic Understand ing published by Science Research Associates, 
Inc.) on a pre-test and post-test basis to determine the extent to which 
teachers increased their knowledge of basic economics, a specially designed 
questionnaire to enable the teachers to evaluate their workshop experiences, 
and evaluation of projects and materials prepared by the Sears Fellows for 
use in their schools (this evaluation being made by economic education 
specialists not involved In the Program), and formal and informal evaluations 
of the teachers by educational administrators (such as principals) in a 
position to observe their efforts to teach economics- The results of this 
Phase of the evaluation (reported separately) indicated that the Soars FelJows 
generally learned about as much in their workshops as college students learn 
in a full-semester course in prim ii)les of economics, that the materials they 
developed were better than might be expected of teachers with similar back- 
grounds and experience, and that the overwhelming majority had made sincere 
and effective efforts to increase and improve the teaching of economics in 
their schools- Phase 'iVo, which will be the subject of this report, was 
designed to answer the question: "To what extent did the workshop training 
o[ the toarhers affect the economic vinderstand I ng of their pupils?" 

The basic approach was to select a group of teachers from among the Sears 
F»,'llows and compare tlie changes In economic understanding achieved by their 
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pupils with ":he changes achieved by Che pupils of similar teachers who did 
not have the workshop training. The Primary Test o^ Economic Understanding 
(PTEU) developed by Donald Davison and John Kilgore at The University of 
Iowa was selected for use at the second, third, and fourth grade levels. The 
PTEU had been administered to over 500 second graders and nearly 5,000 third 
graders; it had a reliability coefficient (Alpha/KR 20) of 0.78 (which is good 
for a test of this length), and norming data were available.^ The authors' 
claim of content validity was based upon their many years of personal contact 
with primary level teachers and a careful examination of textbooks and materials 
developed by school systems. Because the PTEU had not been tried with fourth 
graders, there were some doubts about the wisdom of using it in this eval- 
uation with fourth grade pupils. These doubts proved to be unfounded, how- 
ever. The PTEU was found to have a reliability coefficient of 0.75 when used 
as a pre-test with fourth graders, and 0.80 when used as a posu-test. (The 
recommended reliability coefficient is 0.70.) In short, this test appears to 
have been adequate for purposes of this study. 

The Test of Elementary Economics (TEE) developed by t:he West Springfield, 
Massachusetts St^hool System, was selected for use with pupils in grades five 
and six. This test had been tried with nearly 2,500 pupils in nine school 
systems and norming data were available. We were uncertain, however, tliat the 
norminj; sample was truly representativov and therefore decided to roly most 
heavilv on the PTKU. Indeed, fifth and sixth grade teachers wore in(luded in 
the (*vaJuation initially only because of the fear that we might not get th.e 
20 experimental teachers needed from among those serving in grades two, three, 

^See Donald G. Davison and John H. Kilgore, "A Model for Evalu?iting the 
Effectiveness of Economic Education in Primary Grades," The J o ui nal of Economic 
Ed uca t ion , Fa II, 1971, and Ex aminer^ s Manual: Primary Test of Econ omic Under*- 
standinj»^ (Iowa City: The University of Iowa, 1971) for details. 



and four. However, a reliability coeCficient of 0.77 was obtained at che 
sixth grade level and 0.68 at the fifth grade level when the TEE was used 
as a post-test.* Item analyses suggest that the TEE may not bo as reliable 
as the PTEU, but it does seem to have been adequate for purposes of this 
study, especially in view of the fact that we actually did obtain sufficient 
data from over 20 teachers using- the PTEU.** 

2 

Flanagan's Test of General Ability (TOGA) was selected to determine the 
academic ability of the pupils. The appropriate form for each different grade 
level was to be administered to all pupils in both the experimental and control 
classrooms. Differences in abilicy levels would then be taken into account 
in analyzing pupil change scores on the PTEU or TEE. In the next section, 
the actual setting up of the evaluation program is described and explained. 

Settin R ui3 the Testing; Program 

The original plan wau to select 20 of the Sears Fellows for participation 
in the pupil testing program. These teachers would be known is the experime n tal 
teachers » and their classes would be called experimental crou ps or experimental 
classes. For each experimental teacher and experimental class it was orig- 
inally planned that there would be two control teachers and two control groups. 
C on t rol group numb er one was to be made up of the pupils of teachers who were 
attempting to include economics in the elementary curriculum, but who had not 
attended a sunnier workshop in economic education. The controls were to be in 
different schools in the same district or areas as the exper iTiiontals, they 
would be at the same grade level as the experiraentals, and generally of the 

2 

Published by Science Research Associates, Inc. 
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See Appendix A for reliability coefficients of the PTEU and TEE, 
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See Appendix B for difficulty and discrimination indices for PTKU and 
Appendix C for difficulty and discrimination indices for TEE. 
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same socio-economic background. Control ^roup number tv;o would consist 
of the pupils of teachers who were n ot atle^mpting to Lnclude economics in 
the curriculum, and who had not attended a summer workshop. In every way 
possible, nhis group would be similar to the others. An effort would also 
be made to find teachers who were similar in terms of age, background, and 
experience. The experimental and control teachers would also be of the 
same sex. 

It was realized from the beginning thac a random sample of teachers 
could not be obtained. Instead, a ^'convenience sample" would be used — a 
factor wliich is undesirable but unavoidable under the circumstances. " Of 
necessity, the experimental ceachers would have to be those who could give 
some reasonable assurance that the testing could be carried out in their 
schools and that control teachers and control classes of the typo desired 
could be obtained. It was also considered imperative that the Joint Council' 
'Director of Research meet personally with each experimental teacher. Since 
li was impossible for the Director to go to every workshop which had Sears 
Fellows among the participants, the experimental teachers had tc be selected 
from among those which ho was able to visit. 

During his visits to the workshops, the Director of Research would meet 
privately with the Sears Fellows, obtain information from them about their 
backgrounds , interests , and teaching assignmen ts , and give them a f al rly 
doLailod explanation of the research design. Aftei pointing out that oacli 
experimental teacher would receive an honorarium (going as high as $100), 
ht' would ask for volunteers for the testing program. Of course, the fact 

^Kor example, if. the experimental class was in a small school in a rural 
area» the control classes would also be in small, nu'al schools. Innor-city 
ghetto children would he matched with other jnner-clty glictto classes in 
different schools. 



thai the experinienial leachers wore volunteers present*; the possibility 
of a biased sample, because one might expect these to be educators who were 
more interested in the program and who had a greater feeling of confidence 
in their ability to teach economics. On the other hand, the prospects of 
a fairly generous honorarium might have offset any feeling of insecurity on 
the part of those teachers who had doubts about their abilities to teach 
economics or who were less enthusiastic about the program. Furthermore, 
ant)nymity was promised. The teachers were told that the results of the 
testing would probably be published, but that the identity of each pupil 
and teacher would be kept secret. 

After a teacher had agreed to participate, the Director of Research 
would obtain the name and addresi? of her principal, superintendent, or other 
administra»;or whose permission might be necessary or who ought to be apprised 
oi the project as a matter of courtesy. Detailed letters were then written 
to these individuals, explaining the background and purposes of the project, 
the tests that would be used, and the research design to be employed. 

Wlien the necessary permissions had been obtained, tests were ordered and 
sent to the Individual who .would be responsible for test administration and 
reporting in each area. This might be a district official (such as district 
social studies coordinator or elementary education supervisor), a principal., 
or a teacher not otherwise involved in the evaluation project. Each such 
person received a detailed sat of instructions on administering the tests and 
returning the answer sheets. Repeatedly, everyone involved was warned that 
lUMiher the experimental nor the control teachers should see the economics 
lests before the time came for post-testing. This, of course, was an attempt 
to avoid the possibility that they would "teach to the test.*' Arrangements 
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wore made to iiave someone other Lhnn the experimental and control teachers 
c'idmin ist(*r the economics tests, and each proctor was paid <5 modest fee for 
this service .($10 each test session administered). Aft(*r alJ of the 
testing and reporting had been completed, an effort was made (throuj^h n 
questionnaire or a personal follow-up letter to a responsible person in 
the school district involved) to determine whether or not any teacher had 
indeed examined the economics tests at any time before the post-tests were 
iven. 

Before all of his visits were completed, the Director of Research 
learned that onr part of the research design was not practical. The purpose 
of control group number two was to account for pupil maturation. That is, 
some children may do better on the post-test than on the pre-test although 
they have not had economics instruction, simply because they have matured. 
If we found that three points (for example) had been gained by pupils In 
control group number two, as compared with seven points for those in classes 
receiving economics instruction, we would (all other thin£;s being ev^ual) tend 
to assume that the gain attributable to economics instruction was only four 
points rather than seven — three of the seven-point gain being accounted for 
by maturation. Control group, number two was abandoned for two reasons. First, 
the Director of Research discovered that some teachers include economic concepts 
in their lessons withoul realizing it. Durir.g an interview with the Director, 
one of the Scars Fellows stated that she had been teaching some economics to 
her fourth graders but was not aw.ir<' of It until she attended the workshop. 
The Director examined (line by lino) the social studies textbook this teacher 
had been using and found that it did indeed contain several of the economic 
concepts included in the P ri mary Test of Economic Unde rstanding. Of course. 
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it would have hoen impossible for the Director of Research to examine every 
piece of material being used by every teaclier participating in tlie experiment 
to ascertain tlie extent to whicli those materials contain economics. Second, 
several of tlie teachers planned to confine tlieir teaching of economics to 
sliort units of about tliree weeks — a period tiiat is probably too short for 
maturation to be a factor. 

The expectation that some teachers originally agreeing to participate 
would fail to follow through caused the Director of Research to obtain 
commitments from more than 20 Sears Fellows. This proved to be a wise move, 
for several teaciiers (usually tiirougii no fault of tiieir own) were not able 
to carry on as planned. Not counting the group from Des Moines, 17 Sears 
Fellows and 25 control teachers participated. (For reasons to be explained 
later in this report, some of these had to be dropped from consideration when 
the data were processed). In addition, 22 scliool administrators or otticr 
teachers assisted by proctoring tests or in other ways overseeing portions of 
the projects in tlicir areas. (Again, this excludes Des Moines.) 

Tiie teaciiers in tlie Des Moint»s worksiiop presented a different situation. 
Here, througii a spec Lai arrangement witii the workshop director, eacli of the 
.12 participants would be considered Sears Fellows, and eacii would be involved 
in the testing program. TiiLs greatly increased tiie number of teaciiers tlie 
Joint Council had originally agreed to include in the evaluation component of 
tiie FeJlowsiiip Program. Furthermore, the Des Moines group provided tiie best 
situation from tiie point of view of uniformity and control. Two staff members 

^*Several teachers who had not been asked to participate administered 
tests to their pupils anyway and submit t:ed the results. Althougii tiiese 
sliowed substantial gain scores, they were not accepted for inclusion in the 
study, because they had not followed our procedures, lacked adequate controls, 
and were in other ways of questionable validity. 

ERIC r 
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lit the Oes Moines workshop (Donald DnvLson and John Kilgore) were die authors 
of the Primary Test of Economic Understanding , and they iiad hod considerable 
experience in administering the Lest and analyzing the results, tlicy could 
j^uarantee a uniform way of test administration and reporting, and they were 
engaged in a detailed examination and analysis of tiie social studies textbooks 
that tlie teachers would be using. Tiie entire evaluation sclieme could not be 
based upon the Dos Moines group, however, because it could not be asserted 
witli confidence that they were representative of all the Sears Fellows. Indeed, 
Liie Des Moines workshop was one of the few made up entirely of elementary 
level teacliers, and the approach being used was substantially different from 
that found In other workshops. (The usual method is to give tiie participants 
instruction in basic economics, and then give some sort of lessons or exercises 
in how to teach the subject at tiie pre-college level. At Des Moines, iiowever, 
the teachers examined the textbooks they would be using to discover tiie economics 
content and were then given instruction in the economic concepts they would iie 
expected to teach.). 

Returning to tiie question of the representativeness of the selected 
teachers, there is some reason to believe tiiat the experimental group was 
indeed as heterogeneous and, at least in some respects, as varied in knowledge 
and ability as one might expect to find in a randomly chosen group of teachers. 
I'^or example, wiien the- Des Moines teacliers took the Test of Economic Undcr- 
s ^ and i ng on a pre- and post-test basis, their gain scores ranged from a low of 
/.<*ro to a higii of nine, <ind liioir postal est scaled scores from 17 to M). Tlic- mc^an 
staled score gain, however, w^as siigiitly higher tiian that of tiie Sears Fellows 
as a wiiole (A. 27 as compared with 3.8^4) and tiian that of a group of 376 teachers 
enrolled in economics courses otiier than the workshops (A. 27 as compared witii 
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3.75). Similarly, Che experimental teachers f rom workshops other tlian 
Des Moines had gain scores ranging from one to nine and post-test scaled 
scores ranging from 19-31. Their mean gain score was A. 88 scaled score 
points. The mean post-test scaled score for all experimental teachers 
who had had no previous economics instruction was 23.86 — not significantly 
higher than the 23. AA achieved by all Sears Fellows with no previousi training 
in economics.^ The median and mode were identical (2A.00) for both of these 
groups. Furthermore, the experimental teachers appear to have been about as 
lioterogeneous as the whole group in terms of economic knowledge. Their 
standard deviation was 3.A2 as compared with 3.AA for tlio entire group. The 
average deviation from the mean was 2.77 and 2.68 respectively. As far as 
measurable understanding of economics is concerned, die experimental group 
appears to have been fairly representative of all Sears Fellows wlio liad no 
previous economics. When we compare the experimental teacliers with the group 
of 376 teachers enrolled in economics courses other tlian the workshops, we 
find that their mean scaled score on the post-test was not quite as good, 
for those 376 teachers achieved a mean of 26.00. We do not know, however, 
liow many of those teachers had had previous economics instruction, nor do we 
know how many were secondary level teachers. In any event, it is cJcnr chat 
tiic Ceachers selecced for Che experiment were no becter (in Cerms of measur- 
able economic underscanding) Chan ocher Sear£> Fellows and other teacliers taking 
economics courses for whom data are available. 

Throughout the 1972^-73 school year*, the Director of Researcii kept in 
touch with all educators participating in the evaluation program. Tills was 

5 

An effort was made to conline the experimental group to teacliers who 
had had no economic instruction before attending tlie workshops. A similar, 
but less successful, effort was made in regard to control tonoliers. 
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dono through personal letters, memo rand a . telephone calls, and (in a few 
casds) personal visits. Participants were given stamped envelopes addressed 
to the Director of Research and were told to telephone him collect if Lhey 
had any questions or problems. The participants had been asked to indicate 
when they planned to have the pre-tests and posc-tests administered, and in 
advance of these periods the Director would send reminders and again specify 
the procedures to be followed. After the testing was completed, efforts 
were made to find out if there had been any departures fiom the prescribed 
procedures. Where such departures were found (as in the case of one experi- 
mental teacher who set up control groups in her own school) , and where the 
deviation would have raised questions about the validity of the data, the 
results were omitted from the study. 

We insisted that the Test of General Ability be administered as early 
in the school year as possible, and most teachers complied. Unfortunately, 
aside from the Des Moines group, it was impossible to achieve uniformity in 
the administration of the economics tests because of the wide variety of 
approaches being used. Some teachers integrated economics into the elementary 
curriculum througliout the school year, and thus administered the pre-test earl 
in the fall and the post-test late in the spring. Others confined their 
economics instruction to a single semester and gave the tests accordingly. 
Some taught economics only in units of three or four week duration.^ We also 

6 

Researchers have found that two weeks of instruction (one hour a day) 
is sufficient to provide between-group differences for topics in the social 
studies. See Ned A. Flanders, "A National Coordinated Program of Research on 
Teaching Effectiveness" in How Teachers Make A Difference (Washington: U.S. 
Office of education, 1971), p. 106. 



found a great variety of tuxtbooki3, other reading materials, audiovisual 
aids, and teaching techniques being uced.^ It was not possible t.o control 
these factors. In the next section, we shall examine the results of the 
study in Des Moines. 

The Results in Pes Moines 

As pointed out earlier, the Director of the Des Moines workshop agreed 
to involve all of the participants in the Phase Two evaluation, and promised 
ci better controlled situation than was possible elsewhere. Note, again, 
that the Des Moines teachers did not differ significantly from the Sears 
Fellows as a whole in terms of their economic knowledge. Anotlier positive 
factor in the Des Moines situation was that new social studies textbooks were 
being adopted for the elementary grades and that each contained some economic 
concepts. Three different scries were being adopted. Series "A" was for 
grades one through six; Series "B" was for grades one through three; and 
Series "C" was for grades four through six. Des Moines elementary schools 
were permitted Lo make choices among combinations of textbook series. The 
economic content of the approved series provided the focal point for the 
Des Moines workshop. In Des Moines, then, the text material as well as the 
workshop experience could be taken into consideration. 

Of the 22 teachers attending the Des Moines workshop, 17 participated 
in the Phase Two evaluation.* (IVo were omitted because they were kindergarten 
teachers and no economics test was available at that level; two were given 

^An examination of nearly 1,000 studies analyzing differences between 
media revealed that few resulted in significant improvement in the effective- 
ness of instruction. See Lawrence M. Stolurow, "Learning EnvironmcMUs or 
Rooms For Thought" in How Teachers Make A Difference , pp. 136-137. 

•k 

One teacher had two classes, so that 18 experimental classrooms were 
i nvo Lved . 
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different ashL>»nmenth and could not participate; and a control classroom 
could not be established for another.) Twenty control teachers were selected 
for this study. No control teacher was selected from a scliool housing an 
experimental teacher of the same grade level. Table 1 shows the distribution 
of the 38 experimental and control classrooms by grade level and by textbooks 
used . 



Table 1 

Experimental and Control Classrooms, 
by Grade and Textbook 



Grade 


Text A 
E C 


Text B 
E C 


Text C 
E C 


2 


1 


I 










3 


1 


2 


1 


2 






4 










A 


5 


5 


A 


3 






1 


] 


6 


A 


A 






2 


2 



There were 1,091 pupils in the 38 control and experimental classrooms. 
Because of incomplete, data, however, 175 children were eliminated from the 
study. Thus, a total of 916 pupils were used for the Des Moines evaluation. 
All test materials (the PTEU, the TEE, and the TOGA) were handled only by 
principals or consultants appointed to administer the tests. Teachers were 
not permitted to examine them. 

The criterion variable was the individual change score (pu«L-test score 
minus pre-test score) adjusted for differ<^nccs in ability by the use of the 
Individual TOGA scores as the covariate. Since two different economics tests 
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were used in two blocks of grades, 2 through 4 and 5 through 6, the results 
obtained in each of these blocks will be analyzed separately. 
Grades Two through Four . 

Complete data were gathered on 408 pupils in seven experimental and ten. 
control classrooms for these grades. The data included individual pupil 
test scores on TOGA and PTEU (pre and pest) and were analyzed using Che 

g 

regression analog of the analysis o£ covariance as described by Schilling. 

The individual student's TOGA score was used as the covariate in an attempt 

to remove from the final analysis any effects due to differences in individual 

ability. Since a three-factor analysis (grade x textbook x treatment) was 

not possible because the design was not crossed with respect to textbook 

(I.e. alJ texts were not used at all grades), the data were analyzed in a 

two-factor (block x treatment) design. The blocks used in the design were: 

I) grade two, Text "A"; 2) grade throe. Text "A"; 3) grade three. Text *'B", 

and 4) grade four. Text "C". Fortunately, one test of the simple effect of 

textbook was possible between Block 2 and Block 3. A Scheffi procedure 

9 

for multiple comparisons was used to test the hypothesis that within grade 
three Che "A" and "B'* textbooks are equally effective in increasing economic 
understanding as measured by the criterion instrument. 

^^^^ results are presented in Table 2 for the 17 classrooms in grades 
two through four. TOGA and pre-test and po&t-test means on Che PTEU are shown. 
The mean change score is the difference between Che pre-test and post-test 
PTEU classroom scores. The adjusted classroom mean change scores are the 

C. E. Schilling, The Relationship of Analysis of Variance to Regression . 
Paper presented at the Sixteenth Annual Technical Conference j American Society 
for Quality Control, Knoxville, Tennessee, October 19, 1972. 
9 

A. L. Hays, S tatistics for Psychologists , (Kew York: Holt, Rhinehart 
and Winston, 1963). 
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means by clasbroon^ of tlie individual 'c.liange scores adjusted for individual 
TOGA differences. Adjusted classroom mean change scores, by blocks and by 
treatment, arc presented in Table 3, and the results of the regression 
analog of the analysis of covarlance are reported in Table A. 

As shown by the summary table for the analysis of covariancc in Table A, 
the adjusted mean change score for students studying under workshop teachers 
was significantly higher (at the .01 level of confidence) than for those 
whose teachers had not attended the workshops. This suggests that pjjpils 
study Ing under workiyhop teachers learn more economics as measured by the 
PTKU. This assumes random selection of pupils. 

It was possible in Ulocks 2 and 3 to examine the relative effectiveness 
of Texts *'A" and "B" because both were used at the third grade level. How- 
ever, the Scheffi procedure did not produce a significant difference between 
th'\se two Blocks. The critical difference was 2.128 and the actual difference 
v/as 1.909, indicating no significant ditference in the relative effectiveness 
of the two textbooks at grade three. 

Gr ades Five and Six 

In these two grades, TOGA and TEK data were gathered from 508 students 
in 21 classrooms — 11 experimental and 10 control. TOGA and pre-test and 
post-tost means for the f Itth and sixth grades arc reported in Table 5. The 
adjusted mean change classroom scores by treatment, grade, and text are 
reported In Table 6. The results of the regression analog of the analysis 
of covariance are reported in Table 7. 

It was possible to use a three-factor analysis (grade x t<»xtbook x 
ir(»atment) foi analyzing thefiftli .ui<j sixth grade data because at oarh of 
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Table 3 

Adjusccd Mean Gain Scores 
By Block & By TfeaCmcnt 



Grou p Mci.nn N 
B LOCK 1 

Experimental A. 9772 16 

Control 2.4713 23 

Combined 3.2821 39 

BLOCK 2 

Experimental 2.1046 24 

Control 1.2593 -54 

Combined 1.4872 78 

BLOCK 3 

Experimental 4.1279 17 

Control 3.1765 51 

Combined 3.3971 68 

BLOCK 4 

Experimental 4.2200 100 

Control 3.0163 123 

Combined 3.5561 223 



Table 4 

Analysis of Covariance by Blocks 
for Grades 2-4 



Source , df SS MS F 



Trcntnont 


1 


136.2051 


136. 


2051 


9.123** 


Blocks 


3 


230.2176 


76, 


7392 


5.1A0* 


TriMtciont x Block 




iQ.sa/a 


6. 


5080 


.436 


Within Ccllii 


399 


5.9'i6.Hl«7 


lA. 


929A 




Total (AdJuHCod) 


AO 6 
« 


6-. ■)/.?. 7655 


15. 


6226 




TOGA (CovarlaLf) 


1 




3A. 


,A89A 




Total 


/.07 











p < .01 
* p < .05 



df Dcp,rees of Freedom 
SS « Sum:: of Square 
MS « Moan Square 
F » F-iosc 
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Table 6 

Moan Gain (Adjusted) by Treatment. Grade, & Text 

Grades 5-6 



Grade 



5 
6 



Wot'ks'uo)) 



SERIES A 
5.2818 
4.8803 



N o Workshop 



3.0309 
3.20AA 



5 
6 



SERIES C 
4.1634 
2.9623 



3.1128 
3.8739 



ERIC 
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Table 7 

Analysis of Covariancc 
Grades 5-6 



Source 


df 


SS 


MS 




Trcacraent 


1 


149.5226 


149.5226 


^4.962* 


Text 


1 


46.7085 


46.7085 


i.530 


Grade 


JL 


on c/ 1 r\ 




J . OOo 


Treatment x Text 


1 


92.5203 


■ 92.5203 


3.071 


Troacment: x Grade 


1 


42.9728 


42.9728 


1.426 


Text X Grade 


1 


00,0156 


00.0156 


0.005 


Treatment x Text x Grade 


1 


10.0204 


10.0204 


0.333 


Within Cells 


499 


15,035.4515 


30.1312 




Total (Adjusced) 


506 


15,421.1452 






TOGA (Covarlate) 


1 


380.6245 


380.6245 




Total 


507 


15,801.7697 







* p < .o.s 



df = Degrees of Freedom 
SS = Sums of Square 
MS = Mean Square 
F = F^-test 
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these grades there were bi)Ch experxmental and control classrooms and use 
of each of the textbooks "A" and "C". For grades fiv* and six the F value 
for treatment wa^^ significant at the .05 level, indicating that tlie students 
studying under wo r kshop teachers learned more economics as measured by TEE. 
Again, a random selection of students has been assumed. 

As shown in Table 7, no significant results were found for the text or 
grade. Evidently, no significant difference occurs whether the text used 
Is from either series '*A" or "C" at either grades five or six; and this is 
the c-anc whether the children are in experimental or control classrooms. 

T he Results in Other Areas 

In the areas outside Des Mcines, 17 of the Sears Fellows and 25 of the 
control teachers who had agreed to participate in the study did follow 
through and submit data. Five had to be eliminated, however. One Sears 
Fellow arranged for three control classes in her own school and one control 
class in another school in the same area. The data obtained from the three 
classes in the same school were excluded from the analysis. Two control 
teachers from another area ^*ailed to provide TOGA scores, so their economics 
tests were not included In the study. Table 8 lists the eight workshops 
outside Des Moines from which experimental teachers were obtained, and shows 
the number of experimental and control classes in each of those areas. 

Because there were substantial differences among the workshops, each 
is being analyzed separately. (The workshop courses differed in terms of 
overall length, economic content ct)vered, teaching methods employed, the 
extent to which petlagt)gv a.s well .»s e(c>nc)mics was included, make-up of the 
participants* and the like.) I'ahle 9 Indicates the way in which the 37 
classrooms for which dai.i wviv .in.ily/.ed were distributed by workshop and by 
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Table 8 

Experimental and Control Classrooms 
by Workshop (Grades 2-6) 



CLASSROOM 



WORKSHOP" 


E 


C 


A 


2 


A 


B 


2 


A 


C 


1 


2 


D 


A 


3 


E 


2 


3 


F 


1 


1 


G 


1 


1 


H 


3 


3 


TOTAL 


16 


21 



^Because all teachers were promised anonymity, the workshops they attended 
will be identified only by letter. 
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Table 9 

Kxperimental and ConCrol Classrooms by 
Workshop and by Grade, 2-6 





Grade 


2 


3 


A 


5 


6 


WORKSHOP 


K 


C 


E 


C 


E 


C 


E C 




C 


A 


1 


2 


1 


2 












B 






1 


2 


1 


2 








C 


1 


2 
















D 


i 




1 


2 


2 - 


1 








E 


1 


I 






1 


2 








F 






1 


1 












G 


1 


1 
















II 






I 


] 






i 1 


1 


1 


TOTAL 


5 


6 


5 


8 






I 1 


1 


1 



Information on the sample population * 

*Data gathered on 42 classrooms (5 of A2 classrooms dropped) 

*Data analyzed for 37 clavSsrooms (33 classrooms, grades 2-A 

A classrooms, grades 5-6) 

*1,102 students in A2 classrooms originally involved 

*950 stvidenCs in 37 classrooms retained in the study 

^Complete data on 752 students in 37 classrooms 

*Eight workshops in seven states involved 

-^16 experimental groups, grades 2-6 

*23 rommunifios and 37 schools involved 



ERLC 



^*rnde lovel. The 37 classrooms had a total of 950 pupils, but 198 of 
these were dropped because of Lncowpletc data. (For instance., some took 
the pre-tet^.t bi^t not the post-test, some failed to take the TOGA, and so 
on.) A total of 752 pupils were included in the analysis. Seven states, 
23 communities, and J7 different schools are represented by this population. 
The task of providing I or" uniform procedures and reporting was monumental. 

The t-tests of the difference between treatment group means were cal- 
culated separately for each workshop. Table 10 shows the unadjusted mean 
gain for the entire group outside Des Moines. The pupils of Sears rol.Jows 
achieved a mean gain of 5.23 points, as compared with 2.78 for the pupils 
of control teachers. 



Table 10 

Mean Gain (Unadjusted) By Treatment Group 
Grades 2-A 



Group 


Mean 


SD 


N 


Experimental 


5.23 


5.35 


29A 


Control 


2.78 


A.3A 


36A 


Total 


3.87 


A. 97 


658 



Because of the differences among workshops, and the great differences 
in participating schools and classrooms, the data in Table 11 and Table 12 
are probably more meaningful. Table 11 indicates that, with the exception 
of one workshop, the pupils of the Sears Teachers mddt: significantly greater 
gains. Five of the mean gains were significant at the •Oi (one-tailed) level, 
and two at the .05 (one-tailed) level. Table 12 shows that the gains made 
by fifth and sixth grade pupils of Scars Fellows were significantly greater 
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Table 11 

Summary of t-tests for Grades 2-4 



Workshop 


Experimental 


Control 


L 


Mean 


SD 


N 


Mean 


SD 


N 


A 


4.55 


4 


.75 


38 


1. 


53 


5. 


14 


53 


2. 


98** 


B 


2.11 


3 


.38 


19 


3. 


19 


3. 


80 


21 


-0. 


77 


C 


5. 34 


5 


.27 


41 


2. 


53 


3. 


99 


57 


2. 


26* 


D 


4.00 


5 


.83 


79 


2. 


55 


4. 


37 


60 


1 


7 5* 


F. 


5.81 


5 


.52 


53 


2. 


75 


4. 


44 


96 


3 


93** 


F 


30.46 


8 


.38 


13 


5 


85 


3. 


73 


33 


2 


33** 




4.2/ 


4 


.38 


26 


0 


30 


J . 


50 


2U 


3 


55** 


H 


7.40 


3 


.95 


25 


2 


08 


3. 


94 


24 


5 


27** 



* p < .05 
** p < .01 



Table 12 



Summary of t-tests Cor Grades 5-6 



Workshop 


Experimental 


Control 


t 




Mean SD N 


Mean SD N 




H 


. 7.05 5.3A A3 


2.29 2.66 51 


A. 



** p < .01 
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than those made by control pupils at the .01 level. (This involved only 
one fifth grade experimental and bnt fifth grade control, and one sixth 
grade experimental and one sixth grade control.) 

Although the workshops outside Dcs Moines were very varied, and although 
the researchers could not exercise as much control over test administration 
as desired, the findings generally support those obtained in Des Moines. 
That is> at every grade level t he pu p ils of Sears F e llows tended to achieve 
signifi cantly h ighe r gain scores than those of teachers who had not had 
workshop training, after adjusting for differences in student ability as 
measured by the TOCA.^^ 



As expected', there were many individual differences. In the few instances 
in which control groups achieved greater gains than experimental classes, one 
can only speculate as to the reasons. Table 13 gives a breakdown of the results 
for each experimental teacher having one or more control groups providing usable 
data. Note that the second control group for the third grade experimental 
teacher from Workshop A achieved a gain of 5.043 while the experimental cla^.s 
achieved a mean gain of 2.76A. Follow-up questionnaires to the teachers, and 
communications with an official of the school district, revealed th.at the pupils 
of that control teacher l:ad been exposed to eccnomir,s instruction from kinder- 
garten on up, that the control teacher had examined the PTEU (contrary to 
instruction), that she had had an economics course in college, and that she 
had nearly twice as many years of teaching experience. Any one of these factors 
(probably the first), or some combination of them, might explain the superior 
performzince of her pupils. (It was also reported that those pupils were of 
iiigher ac.ademic ability and had higher reading scores.) Moving to Workshop B, 
wc find that the control class achieved a slightly higher gain score, although 
this was not statii>Lieally significant. Follow-up study revealed no explana- 
tion for the failure of the experimental group to do better, except for tl:e 
possibility that they were less talented academically. There are few clues 
a,s to wliy the fourth grade experimental teacher trom Workshop C got poorer 
results than her second control teacher. Perhaps the fact that the control 
teacher taught economics as a separate unit while the experimental teachei 
attempted to integrate economics in the regular curriculum was a factor, 
er perhaps it was because the control teacher had had 11 years of experience 
as compared with only one year for the Sears Fellow. The situation in 
regard to the third grade teacher from Workshop D cannot he explained. 
F.xcept for the fact that the set ond ct)ntrol group teacher had had economics 
in college and a bit more experience than both the experimental teacher and 
the first control teacher, these classes and teachers were almost perfectly 
matched. in any event, these few exceptions (although they raise tantalizing 
questions — as do nearly all research results) do not negate the overall 
result — the pupils of workshop teachers generally do significantly better. 
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Conclusions and Recommendcitions 

As Rosenshine points out> tliere has been a "paucity of classroom 

research on how teachers make a difference." He notes that of the 1,000 

papers presented at the 1971 meeting of the American Educational Research 

Association, no more than 15 focused on how teachers make a difference on 

measured by pupil gains. In the area of economic education, we know of 

only three studies which have attempted to measure r.he impact of the 

teacher's workshop training on his or her pupils, and these were extremely 
12 

limited in scope. In a sense, then, this study represents a pioneering 
effort. 

Flanders has stated that: "The consequences of teaching and learning 
can never be completely determined and attempts to measure them are merely 
estimates based upon partial information." Nevertheless, he goes on to 
suggest that data should be collected "In order to make the besL possible 
guess about whether more was luarnod in one treatment compared with the 
other." These statements apply to our study, and it is recognized from 
the beginning that this attempt at evaluating the impact of economics 
education workshops would encounter many difficulties, provide only partial 
information, and leave many questions unanswered. The inability to exercise 

Barak Rosenshine, "New Directions for Research on Teaching," How 
Teachers Mad e a Difference , p. 67. Also see his statements on pp. 68-73. 

12 

See Robert J. Highsmitli, A Study to Measure the Imi^act of In-Service 
Institutes on the Students of Teachers Who Have Participated. Unpublished 
paper. St. Cloud, Minnesota: St. Cloud State College, 1971. Also, Dennis 
M. 0* Toole, Evaluation of an Economics Institute for Secondary School Teachers. 
Unpublished paper. Richmond, Virginia: Virginia Council on Economic Education, 
1971. A third study, unpublished and untitled, was made by William Luker of 
the Texas Council on Economic Education, Denton, Texas, in 1971. 

13 

In How Te ac hers Make a Difference » p. 110. 
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complete control over r.est administration outside Des Moines > and the lack 
of complete information c?n sucli things as differences in teaching techniques 
employed in the various classrooms> are inherent weaknesses. 

As measured by the Primary Test of Economic Understanding (PTEU) and 
the Test of Elementary Economics (TEE), the students of workshop teachers 
did significantly better than those of control teachers who had not attended 
the workshops. This was the case after scores had been adjusted for differences 
in pupil ability as measured by the Flanagan Test of General Ability (TOGA), 
T!ie findings apply to every grade level from two through six. Tn the 
Des Moines situation, where pupil textbooks being used were also considered, 
no textbook \/as found to be better than any other in terms of pupil gains on 
the economics tests. Outside Des Moines, so many different books and materials 
were being used by the experimental and control classes that it was not pos- 
sible to analyze their impact. 

Of course, one cannot overlook the possibility that other conditions 
might have had some bearing on pupil achievement. It is possible that the 
Sears Fellows did better because they were more capable teachers than the 
controls, and that they had these capabilitiecs even before their workshop 
training. After ail, they were selecLeJ from among several hundred teachers 
who applied for the grants, and their records were considered during the 
selection process. On the other hand, those picked for participation in the 
study were found to be no better in terms of economic understanding (as measured 
by the Test of Economic Understanding published by Science Research Associates, 
Inc.) than the group of Sears Fellows as a whole, and there is some fra^;mentary 
evidence that they were no better in this respect than other teachers who have 
completed workshop.s or economir.s iuurscs. The Des Moines teachers did not 

i 

I I 
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appear to be better than the average teacher from that area, accoirding to 
members of the workshop staff who base this judgment on many years of experi- 
ence with Des Moines teachers in workshops and in daily classroom contacts. 
In short, the researchers did not "stack the cards" by choosing teachers 
known to be more capable than their colleagues. 

Perhaps the experimental teachers were more highly motivated than the 
control teachers. The fact that they gave up part of their summer vacations 
to attend workshops might be considered evidence of motivation and interest 
in economics, but this could be offset by the possibility that the money 
offered to Sears Fellows served as a primary inducement. (Several teachers 
interviewed by the Director of Research stated that^ the money rather than an 
inherent interest of economics served as the motivation to attend, or at 
least to attend the economic education workshop ratiier than some other.) 
In the questionnaire sent to al] teachers (control as well as experimental) 
after all the testing and reporting had been completed, every respondent 
indicated tlie belief that economics can and should be taught at the elementary 
level. Whatever their interest and motivation prior to attending the workshop, 
however, most Sears Fellows gave evidence that the experience served to convince 
them of the need for economic education at all levels. Unfortunately, there is 
simply no way of comparing the motivation of the Sears Fellows with that of 
the control teachers. 

Tlie 50-called "Hawthorne Effect" should also be taken into account. The 
Sears Fellows knew they were part of an experiment and that their pupils were 
being pre- and poot-tested. This, alone, might have induced them to make an 
unusually strong effort to teach economics. On the other hand, the control 
teachers also knew that they were part of an experiment, tl>3t their pupils 



I 



I 
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vere being cesced, and Chat the results would be compared with chose of the 

experimental classrooms. Thus, if the Hawthorne Effect was indeed a factor, 

it applied to both the experimental and control groups and therefore shcfuld - 

lA 

not nullify the results of this study. 

Other things that might have accounted for the superior performance of 
the experimental pupils, at least in part, are the methods used by the Sears 
Foundation Fellows, better rapport between pupils and teachers, more support 
from the school administration, follow-up assistance provided by the workshop 
staff, and the use of materials given to Sears Foundation Fellows but not to 
control teachers. In view of the large number of teachers Involved, however, 
it is probable that these factors applied in some cases but not in others, 
and that any such factor prevailing in one situation was balanced by its being 
absent in another. (For example, some Sears Foundation Fellows reported 
that their administrators becanv^. very much interested in economic education 
and provided considerable support, while others stated that chey were unable 
to nrouf*** the interest of anyone else in their schools or systems-) It must 
also be taken into account that where such factors as the use of more creative 
teaching methods, better administrative support, and follow-up assistance from 
economic education specialists existed, they might have been the results of 
the follow-up assistance, a service that was promised to each recipient of the 
Soars Foundation Fellowships. 

■^^ControHed experiments designed to determine whether or not the 
Hawthorne Effect is a significant factor in educational research involving 
elementary pupils have failed to show that it has any Impact. See Desmond 
Cook, T he Impact of the Hawthorne Kffect on Experimental Desi)^ns in E ducational 
Resjivar^h (Columbus: The Ohio State University, 1967), and Robert W. Bauernfeind 
and Carl J. Olson, "Is the Hawthorne Effect in Educational Experiments a 
Chimera?" Phi Delta Kappan, December, 1973, pp. 271-273. 
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The follow-up study nmde after all testing and reporting had been 
completed revealed that the teachers who attended the workshops were more 
likely no make a conscious effort to include economics in the curriculum, 
and chat they felt more adequately prepared to teach economics than did 
thfi control teachers. Of course, this is not surprising. It was also 
found that the workshop teachers were more likely to present economics 
as a separate unit as opposed to (or in addition to) integrating the 
concepts into the regular elementary curriculum* 

Although we can conclude with considerable confidence that the training 
received by the elementary teachers in economic education workshops had a 
significant Impact on their pupils, several recommendations should be consid- 
ered if similar studies are to be made in the futuie. Future studies might 
be undertaken only if pore adequate controls are possible. The well- 
controlled Des Moines situation as described earlier in this report .should 
he the rule rather than the exception. Further research is needed on the 
possible effects of textbooks and other materials. Although this study 
found that one textbook series was just as effective as any other being used 
in the Des Moines situation, it must be realized that those materials had 
just been rdopCed. It is possible that after a year or two of experience 
with the new materials, some difference might be found. (That is, other 
things bein« equals one textbook might prove to be more effective than another 

in improving the economic knowledge oC the pupils.) The books used in 

15 

Des Moines were relatively good in terms of economic content- A future 
scudy migiit compare the impact of these materials with those known to have 
loss economics content. 

i 5 

See Donald G, Davison, John Kilgore, and Larry C. Sp^ontz, Kconomics 
ll' So cial Studies T extbooks: An Evnluaticn of the Economics and the Teaching 
Krral05>ie s in Social Studies Textb ooks , Elementary Grad e s (1-6 ). (New York: 
TLo Joint Council on Economic Education, 1973.) 
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Research is needed to determine whether or not a given method of 
teaching is superior to other methods. For exarnple, do children using 
economics games and simulations learn more than those being taught by 
more traditional methods? Is it better to teach economics in separate 
units, or do pupils learn as much if the same content is integrated in 
the elementary curriculum throughout the semester or school year? To what 
_ extent do children retain what they have learned? How much will they 

remember a year or two later? 

Does the design o£ the workshop make a difference? Is it better to 
stt'ess teaching techniques and materials, as v/as done in Des Moines, or to 
continue with the traditional approach of stressing economic principles 
(as in a regular college introductory course) with pedagogy as a secondary 
consideration.^^ Farther, are workshops more effective if they contain 
elementary level teachers only, or is Che usual procedure of mixing eleraen- 
' tary with secondary teachers just as good? 

To what extent does teacher knowledge of economics make a difference? 
Other things being equal, does the teacher who achieved a high score on 
the Tes t of E conomic Understanding (or some other standardized instrument) 
get better results than the teachei who^e score was low? (An attempt was 
made to answer this question in this study, but the data were not adequate 
to justify a firm conclusion.) 

^^^\lthough very limited in scope, two recent studies suggest that teachers 
achieve higher gain scores on a standardized economics test when they take a 
course especially designed for elementary teachers in which instruction in 
how Lo teach economics in the elementary grades is combined with instruction 
in basic economic principles. See Loren Guffey and Charmayne Cullom, A Note 
on Increasing Economic Understanding. Unpublished paper. (Conway, Arkansas: 
State College of Arkansas, 1973) and Dennis 0*Toole and Ann Coates, An Experi- 
ment in Open Economics* Unpublished paper. (Richmond: Virginia Commonwealth 
University, 1973) • 

ERLC 
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Do children do better if they have had an organized* sequential exposure 
to economics from first grade (or kindergarten) on up? Or is it just as 
effective to concentrate economics at one grade level? Intuitively, we 
would be inclined to choose the former, and some fragmentary evidence from 
this study supports that view. Nevertheless, controlled research is needed 
on this point. Furthermore, if pupils are given a sequential exposure to 
economics over a period of years, what concepts should be taught at each 
grade level? What is the best scope and sequence? 

How important is administrative support, or a centrally guided economics 

program? Will beLLer results be obtained if there is a school-wide or 

district economics program, or can a teacher do just as well working in 
] 7 

isolation. These and many other questions call for research. This 
evaluation confirms the many previous studies which have shown that elementary 
level children can indeed learn some basic economic concepts. It suggests 
very strongly that workshop training for teachers does "pay off" in terms of 
improved pupil learning. But, like nearly all educational research, it 
raises more questions than it answers. If we now "know" that children can 
Jcarn economics, and that economic education workshops are indeed effectivp 
in increasing the economic understanding of the pupils of the participants, 
we still need to know more about how to improve the existing procedures and 
practices. 



Some of the Sears Foundation Fellows whose pupils achieved the greatest 
gain scores reported that they had been able to "turn on" their colleagues and 
get good administrative support. Some affected their entire districts as well 
as their own schools. 
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Appendix A 



Reliability CoetficienCs 



PRE-PTEU 

Rellabillcy 

Grade Obtained Recommended Standard Error Measureme nt 

2 .69 . .70 2.32 

3 .80 .70 2.A5 
A .75 .70 2.AA 

POST-PTEU 

2 .72 .70 2.49 

3 .75 .70 2.A8 
A .80 .70 2.25 



PRE-TEE 

Reliability 

Grade Obtained Recommende d S tandard Error Measurement 

5 .58 .70 2.82 

6 .63 .70 2.93 

POST-TEE 

5 .68 .70 2.92 

6 .77 .70 2.89 



The re Liability used In this study was derived by use of the Kuder 
Richardson Formula 20. 

PTEU data obtained from 17 clat^srooms in Des Moines. 
TEE data obtained from 21 classrooms in Des Moines. 
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